Papers with Reward models
M-RewardBench: Evaluating Reward Models in Multilingual Settings (2025.acl-long)
Copied to clipboard
Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee
| Challenge: | Reward models (RMs) are primarily trained and evaluated in English and their capabilities in multilingual settings remain understudied. |
| Approach: | They construct a multilingual RM evaluation benchmark that tests the chat, safety, reasoning, and translation capabilities of RMs in 23 languages. |
| Outcome: | The proposed model performs better for high-resource languages and improves with translation quality. |
RELIC: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples (2025.findings-emnlp)
Copied to clipboard
Soumya Suvra Ghosal, Vaibhav Singh, Akash Ghosh, Soumyabrata Pal, Subhadip Baidya, Sriparna Saha, Dinesh Manocha
| Challenge: | a new reward model for low-resource Indic languages is proposed . a preference-based training approach is prohibitively expensive, authors say . |
| Approach: | a new in-context learning framework is proposed to train a retriever to select in-constext examples from low-resource Indic languages. |
| Outcome: | a new in-context learning framework for reward modeling in low-resource Indic languages is developed . the proposed framework outperforms existing examples on three preference datasets . |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited by their large-scale annotation and prone to reward overoptimization. |
| Approach: | They propose a training paradigm that integrates three complementary strategies to address these challenges by reformulating question–answer pairs into preference-task instructions, averaging the rewards aggregated from diverse preference- task instructions for each sample, and a balancing outputs from the value head under different dropout rates. |
| Outcome: | Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization. |
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reward models have a high performance on benchmarks, but performance degradation is often due to overfitting. |
| Approach: | They propose to explicitly train reward models to assign similar scores to paraphrases to improve their robustness. |
| Outcome: | The proposed model reduces degradation by half for the Chat Hard subset in RewardBench. |
Improving Reward Models with Synthetic Critiques (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that reward models overfit on superficial features, hindering generalization performance . prevailing approach to training preference-based reward models presents several challenges . |
| Approach: | They propose a method that uses synthetic natural language critiques to provide additional feedback to large language models. |
| Outcome: | The proposed approach improves performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. |
Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing failure discovery methods rely on prior knowledge of preference attributes . Existing methods do not scale to new models or data. |
| Approach: | They propose a preference distribution agnostic procedure that uses the reward model itself to guide controlled decoding toward mis specified responses while preserving the underlying preference class. |
| Outcome: | The proposed procedure improves robustness without degrading reward quality across models. |
ToolRM: Towards Agentic Tool-Use Reward Modeling (2026.findings-acl)
Copied to clipboard
Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F. Wong, Junyang Lin, Min Yang
| Challenge: | lack of reliable reward models for tool-use tasks has limited progress toward agentic AI . recent advances in agentic artificial intelligence are driven by tool-using capabilities of large language models. |
| Approach: | They propose a pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling to build lightweight reward models. |
| Outcome: | The proposed model outperforms existing models on tool calling tasks with higher accuracy. |
Semi-Supervised Reward Modeling via Iterative Self-Training (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reward models capture values and preferences of humans and are used in Reinforcement Learning with Human Feedback (RLHF) Traditionally, training large language models relies on extensive human-annotated preference data, which poses significant challenges in terms of scalability and cost. |
| Approach: | They propose a method that enhances RM training using unlabeled data. |
| Outcome: | The proposed approach improves reward models without incurring additional labeling costs on unlabeled datasets. |
Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for reward models show a weak correlation with performance of optimized policies . existing benchmarks do not accurately assess the true capabilities of reward models . |
| Approach: | They explore how reward overoptimization captures how well a reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy. |
| Outcome: | The proposed benchmarks show that reward overoptimization is a weak factor . the high correlation with degree of overoptimalization leads to lower correlation with downstream performance . |
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch (2025.acl-long)
Copied to clipboard
Xueru Wen, Jie Lou, Zichao Li, Yaojie Lu, XingYu XingYu, Yuqiu Ji, Guohai Xu, Hongyu Lin, Ben He, Xianpei Han, Le Sun, Debing Zhang
| Challenge: | Existing Chinese resources are small in scale and limited to specific domains, making them insufficient for LLM post-training. |
| Approach: | They propose a Chinese-annotated reward model and a preference dataset to address this gap . they evaluate Chinese RMs on CheemsBench and construct an RM that captures human preferences . |
| Outcome: | The proposed RM achieves state-of-the-art performance on CheemsBench and CheeMePreference. |
Reward Model Perspectives: Whose Opinions Do Reward Models Reward? (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that reward models are poorly aligned with demographic groups and can reward harmful stereotypes. |
| Approach: | They propose a framework for measuring the alignment of opinions captured by RMs . they also investigate the extent to which RM's demonstrate sociodemographic biases a . |
| Outcome: | The proposed framework measures the alignment of opinions captured by RMs . it shows that RM models are poorly aligned with several demographic groups . the findings highlight the need for more careful consideration of RM behavior in model alignment . |
Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems (2025.acl-long)
Copied to clipboard
| Challenge: | Existing reward models focus on human preferences, neglecting verifiable correctness signals. |
| Approach: | They propose a reward system that combines human preference rewards with verifiable correctness signals to provide reliable rewards. |
| Outcome: | The proposed reward agent significantly outperforms vanilla reward models on benchmarks and inference-time best-of-n searches on real-world tasks. |
WildReward: Learning Reward Models from In-the-Wild Human Interactions (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on collecting preference pairs, requiring substantial annotation efforts. |
| Approach: | They propose a pipeline to extract reliable human feedback from in-the-wild interactions . they propose to use WildChat as an interaction source to train the model . |
| Outcome: | The proposed model achieves comparable or even superior performance compared to conventional models with improved calibration and cross-sample consistency. |
PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reward models lack generative and reasoning capabilities, resulting in poor performance. |
| Approach: | They propose a reward-aware task-adaptive reward model that enables pointwise training using readily available pairwise data via a novel Preference-Aware Reward mechanism. |
| Outcome: | The proposed reward model achieves an average relative improvement of 8.7% over the base models on RewardBench and RMBench. |
Debiasing Reward Models via Causally Motivated Inference-Time Intervention (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for mitigating spurious features in RMs focus on response length . Existing methods focus on RM activation, resulting in performance trade-offs . |
| Approach: | They propose a method that uses neurons to suppress spurious features in RMs at inference time. |
| Outcome: | The proposed method reduces sensitivity to spurious features without inducing performance trade-offs on RM benchmarks. |
Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited in generalizability due to distribution shift, preference label noise, and mismatch of challenging samples with model capacity. |
| Approach: | They propose a framework that constructs preference pairs with varying difficulty levels and then produces a specific curriculum for reward model training. |
| Outcome: | The proposed framework improves generalizability of reward models by a significant margin without incurring additional inference costs compared to existing non-curriculum baselines. |
DARM: Distribution-Aware Reward Modeling by Alleviating Biases from Low Preference-Context Dependency Data (2026.acl-long)
Copied to clipboard
Shaofan Liu, Guoqiang Zhang, Shihan Dou, Huiyuan Zheng, Yiming Zhou, Junjie Ye, Shaowen Wang, Shichun Liu, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing methods for training reward models are vulnerable to context neglect and degraded accuracy. |
| Approach: | They propose distribution-aware reward modeling that augments the RM objective with a conditional mutual information regularizer that maximizes context and the predicted reward conditioned on the response. |
| Outcome: | The proposed model improves performance in RLHF and improves accuracy in other settings. |